Sound-picture synchronous compression and synchronous reproduction system

ABSTRACT

In a sound-picture synchronous compression device, the picture compression process is performed on a designated amount of picture codes. Then, a necessary amount of sound codes is calculated on the basis of a time of the picture compression process and is subjected to sound compression process. Consequently, compressed picture codes and compressed sound codes are transmitted to a sound-picture synchronous reproduction device. Herein, the picture reproduction process is performed on a designated amount of the compressed picture codes. Then, a necessary amount of the compressed sound codes is calculated on the basis of a time of the picture reproduction process and is subjected to sound reproduction process. Thus, the device reproduces the picture codes and sound codes, based on which pictures and sounds are output by the monitor and speaker respectively. The switching between the sound processing and picture processing in compression and reproduction is made with a reduced overhead, while the switching is performed in such a way that the sound processing does not break off. Incidentally, the picture codes correspond to a number of macro blocks while the sound codes correspond to a number of AAUs.

This application is a divisional of U.S. Ser. No. 09/022,540 filed Feb.12, 1998, U.S. Pat. No. 6,104,756.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to sound-picture synchronous compression andsynchronous reproduction systems in which sound information and pictureinformation are subjected to synchronous compression and synchronousreproduction in a real-time manner. Specifically, the sound-picturesynchronous compression and synchronous reproduction system(hereinafter, simply referred to as a sound-picture synchronouscompression-reproduction system) is actualized using a sound-picturesynchronous compression device and a sound-picture synchronousreproduction device. This invention is based on patent application No.Hei 9-27140 filed in Japan, the content of which is incorporated hereinby reference.

2. Prior Art

The sound-picture synchronous compression-reproduction system isactualized by a pair of sound-picture synchronous compression device andsound-picture synchronous reproduction which use CPUs (an abbreviationfor “Central Processing Unit”) as well as DSPs (an abbreviation for“Digital Signal Processor”). In the sound-picture synchronouscompression device, sound information and picture information aresubjected to synchronous compression in real time. So, the compressedsound information and compressed picture information are transmitted viatransmission lines to the sound-picture synchronous reproduction device,wherein they are subjected to synchronous reproduction in real time.

Normally, sounds and pictures are transmitted in a digitized manner viatelephone lines and are then recorded on recording media by usingcompressive encoding because an amount of data thereof is enormous. Thecompression technique for the pictures frequently uses an encodingmethod designed on the basis of the DCT (an abbreviation for “DiscreteCosine Transform”), which uses a property that spatial frequencies ofthe pictures tend to be concentrated into low-frequency ranges. Thisencoding method is employed by the international standard, namely, MPEG(an abbreviation for “Motion Picture Experts Group”) Video,recommendation H. 263.

In addition, the compression technique for the sounds frequently uses aso-called perception encoding method which omits inaudible information,which cannot be heard by human ears, by using auditory psychology. Thisencoding method is employed by the international standard, namely, MPEGAudio, recommendation G. 723.

FIG. 10A to FIG. 10F show hierarchy for encoding formats based on thestandard of MPEG Video. Codes of the MPEG Video are assigned to levelsof a hierarchical structure as shown in FIG. 10A to FIG. 10F. A toplevel of hierarchy corresponds to a video sequence which is configuredby multiple GOPs (an abbreviation for “Group Of Picture”). Each GOP isconfigured by multiple pictures.

There are provided three kinds of pictures, namely, I picture, P pictureand B picture. Herein, the I picture corresponds to intra-frame codes;the P picture corresponds to interframe codes with respect to a forwarddirection only; and the B picture correspond to interframe codes withrespect to both of forward and backward directions. The picture consistsof multiple slices corresponding to regions which are determined bypartitioning an overall area of the picture. Each slice is configured bymultiple macro blocks which are arranged from the right to the left orfrom the top to the bottom.

A block of 16×16 dots (or pixels) is divided into blocks of 8×8 dots,which are used for representation of luminance components (i.e., Y1, Y2,Y3 and Y4). In addition, blocks of 8×8 dots are used for representationof chrominance components (i.e., Cb, Cr) in connection with the regionof the luminance components. Thus, the macro block is constructed by sixblocks of 8×8 dots in total. Therefore, the block of 8×8 dots is used asa minimum unit for encoding.

FIG. 11 shows an example of the hierarchy for encoding formats based onthe standard of MPEG Audio. Codes of the MPEG Audio are constructedusing “AAU” (an abbreviation for “Audio Access Unit”), which is aminimum unit for encoding.

Now, a description will be given with respect to the conventionaltechnique for compression of sound codes and picture codes. FIG. 12 is ablock diagram showing an example of a sound-picture synchronouscompression device which performs synchronous compression andtransmission on sounds and pictures. In FIG. 12, a CPU 2 executesprograms of a device control block 4 so as to control the device as awhole. A user operates a keyboard 3 to input commands and instructionsto the device. States of execution of the programs are visuallydisplayed on a screen of a display 1.

Picture data are taken by a camera 6 and are input to the device bymeans of a picture input block 5. Sound data are created by a microphone8 and are input to the device by means of a sound input block 7. Thesound data are subjected to compression by a sound compression block 10.The picture data are subjected to compression by a picture compressionblock 9. The compressed sound code corresponding to the compressed sounddata as well as the compressed picture code corresponding to thecompressed picture data are mixed together using a single code by a codenixing block 11. Then, mixed codes are transmitted onto a transmissionline by a transmitter block 12 as well as a modem 13.

FIG. 13 is a block diagram showing an example of a sound-picturesynchronous reproduction device which performs receiving and synchronousreproduction on sound-picture codes transmitted thereto. In FIG. 13, aCPU 22 executes programs of a device control block 24 to control thedevice as a whole. A user operates a keyboard 23 to input commands andinstructions to the device. States of execution of the programs arevisually displayed on a screen of a display 21.

A receiver block 32 receives mixed codes transmitted thereto from amodem 33 as sound codes and picture codes to be reproduced. Then, a codeseparation block 31 separates the mixed codes into sound codes andpicture codes respectively. The sound codes are subjected to expansion(or reproduction) by a sound reproduction block 30. The picture codesare subjected to reproduction by a picture reproduction block 29. Then,contents of picture data corresponding to the reproduced picture codesare visually displayed on a screen of a monitor 26 by a picture outputblock 25. A sound output block 27 provides a speaker 28 with sound datacorresponding to the reproduced sound codes. So, the speaker 28 producesthe corresponding sounds.

The aforementioned example of the system for sound-picture compressionand reproduction performs communications of codes. Herein, dataprocessing should be performed in a real-time manner. In general, anumber of operations (or calculations) are required in processing forthe compression and reproduction. To enable real-time processing, theconventional system which uses the software for compression andreproduction of sounds and pictures reduces a number of framesrepresenting pictures which are subjected to processing. Herein,parallel operation instructions of the CPU and DSP are used to performmultiple operations at once, so it is possible to accomplish high-speedperformance in the above processing.

FIG. 14 shows an example of the parallel operation instructions. Herein,a 64-bit register R0 stores 4-word values a3, a2, a1 and a0, while aregister R1 stores 4-word values b3, b2, b1 and b0. Those 4-word valuesare added together, so addition results are stored in the register R0.According to the parallel operation instructions described above,addition is performed with respect to four words at once, so it ispossible to perform high-speed processing.

Functions of real-time OS (an abbreviation for “Operating System”) canbe used for the compression and reproduction of the sounds and pictures.Herein, if the sound processing and picture processing are switched overperiodically by a certain period of time, it is possible to performreal-time processing with ease. At a time to switch over the processing,contents of the registers which are presently used should be retained.For this reason, the system performs processing in parallel, andimplements the parallel operation instructions requiring “large-bits”registers each having a large number of bits. In that case, theswitching of the large-bits registers frequently occurs, so it isimpossible to perform high-speed processing. To avoid such a problem, itis necessary to add an additional process which initiates the switchingof the processing between the sound compression and picture compressionat the appropriate timing so that one of them is selectively performed.

It is possible to list some papers which disclose conventional examplesof the system which relates to synchronous processing of the sound andpicture.

For example, the paper of Japanese Patent Laid-Open Publication No.7-64730 (denoted by “paper 1”) discloses a conventional example of thesystem which processes sounds and pictures in a synchronized manner. Thecontent of the paper 1 is summarized as follows:

A buffer memory is normally placed in a data accumulated state so thatit is normally filled with a certain amount of data or more. Areproduction portion of one side always issues a data transfer request,so that data are transferred to reproduction portions of both sides by aunit of a data block.

The paper of Japanese Patent Laid-Open Publication No. 7-75059 (denotedby “paper 2”) discloses a conventional example of the system, thecontent of which is summarized as follows:

The system checks a difference between a reproduction time of the soundand a reproduction time of the picture. If the sound is delayed from thepicture in reproduction, a same screen image is continuously displayed.If the picture is delayed from the sound in reproduction, interpolationis performed with respect to the sound.

The paper of Japanese Patent Laid-Open Publication No. 7-110756 (denotedby “paper 3”) discloses a conventional example of the system, thecontent of which is summarized as follows:

The system checks a time required for reproduction of one frame. Then,the system sets and processes a picture which should be displayed for anext frame. So, the system reproduces the sound accompanied with thepicture.

The paper of Japanese Patent Laid-Open Publication No. 7-184143discloses a conventional example of the system, the content of which issummarized as follows:

The processing of the system is controlled to be suited to the “long”processing whose processing period is long. Herein, after performing thelong processing whose processing time is long, the system performs the“short” processing whose processing time is short multiple times by aconstant period of time.

The aforementioned examples of the sound-picture synchronouscompression-reproduction system may have a large interval of time forswitching over the sound processing and picture processing. In such acase, the system cannot catch up with the timing of execution of thesound processing. In the case where the sound and picture operate inparallel, an overhead for switching over the sound processing andpicture processing becomes large. For this reason, the conventionalsystem suffers from a problem that it cannot catch up with the timing ofexecution of the sound processing, so the reproduction breaks off. Inthe case where adjustment is performed on the switching timing betweenthe sound processing and picture processing in response to theperformance of the CPU and DSP as well as the transmission speed of thecommunication lines, it is necessary to modify the system with respectto the sound processing and picture processing. In that case, there is aproblem that adjustments of the programs take much time.

SUMMARY OF THE INVENTION

It is an object of the invention to provide a sound-picture synchronouscompression-reproduction system which is capable of reducing an overheadat the switching of parallel operation registers and which is alsocapable of realizing the switching of the sound processing by aninterval of time in such a way that the sound processing does not breakoff.

It is another object of the invention to provide a sound-picturesynchronous compression-reproduction system which is capable ofadjusting the switching timing between the sound processing and pictureprocessing without a substantial modification of programs used forcompression and reproduction of the sound and picture.

According to an aspect of the invention, the sound-picture synchronouscompression-reproduction system is actualized by a sound-picturesynchronous compression device and a sound-picture synchronousreproduction device.

In the sound-picture synchronous compression device, the picturecompression process is performed on a designated amount of picturecodes. Then, a necessary amount of sound codes is calculated on thebasis of a time of the picture compression process and is subjected tosound compression process. Herein, if an untransmitted amount ofcompressed sound codes is less than a threshold value, the devicecalculates an amount of sound codes for sound compression by which anamount of the untransmitted compressed sound codes are made greater thanthe threshold value. Thus, the calculated amount of sound codes issubjected to sound compression process. So, compressed picture codes andcompressed sound codes are transmitted to the sound-picture synchronousreproduction device.

In the sound-picture synchronous reproduction device, the picturereproduction process is performed on a designated amount of thecompressed picture codes. Then, a necessary amount of the compressedsound codes is calculated on the basis of a time of the picturereproduction process and is subjected to sound reproduction process.Herein, if an unreproduced amount of the compressed sound codes isgreater than a threshold value, the device calculates an amount ofcompressed sound codes for sound reproduction by which an amount of thecompressed sound codes received are made less than the threshold value.So, the calculated amount of the compressed sound codes is subjected tosound reproduction process. Thus, the device reproduces the picturecodes and sound codes, based on which pictures and sounds are output bythe monitor and speaker respectively.

Incidentally, the picture codes correspond to a number of macro blockswhile the sound codes correspond to a number of AAUs.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects of the subject invention will become more fullyapparent as the following description is read in light of the attacheddrawings wherein:

FIG. 1 is a block diagram showing a configuration of a sound-picturesynchronous compression device in accordance with the embodiment of theinvention;

FIG. 2 is a block diagram showing a configuration of a sound-picturesynchronous reproduction device in accordance with the embodiment of theinvention;

FIG. 3 is a flowchart showing a flow of steps which are performed inconnection with the device of FIG. 1;

FIG. 4 is a flowchart showing a flow of steps which are performed inconnection with the device of FIG. 2;

FIG. 5A is a flowchart showing a routine of compression/reproductioncontrol;

FIG. 5B is a flowchart showing a routine of picturecompression/reproduction;

FIG. 6 is a flowchart showing a routine of compression/reproductioncontrol;

FIG. 7 is a flowchart showing a routine of picture compressionreproduction;

FIG. 8 is a flowchart showing a subroutine of soundcompression/reproduction;

FIG. 9A is a flowchart showing a subroutine for call 1 of the soundcompression/reproduction;

FIG. 9B is a flowchart showing a subroutine for call 2 of the soundcompression/reproduction;

FIGS. 10A, 10B, 10C, 10D, 10E and 10F show an example of hierarchy forcode formats based on the MPEG Video;

FIG. 11 shows an example of hierarchy for code formats based on the MPEGAudio;

FIG. 12 is a block diagram showing an example of the sound-picturesynchronous compression device;

FIG. 13 is a block diagram showing an example of the sound-picturesynchronous reproduction device; and

FIG. 14 shows an example of parallel operation instructions.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a configuration of a sound-picture synchronous compressiondevice in accordance with the embodiment of the invention, wherein partsequivalent to those of FIG. 12 are designated by the same numerals.

Like the aforementioned device of FIG. 12, the device of FIG. 1 isconfigured by the device control block 4 for controlling the device as awhole, the CPU 2 for executing programs, the keyboard 3 for inputtingcommands and instructions given by the user, and the display 1 forvisually displaying states of execution of the programs. In addition,the device of FIG. 1 contains the camera 6 and the picture input block 5which are used to input pictures as well as the microphone 8 and thesound input block 7 which are used to input sounds. Further, the devicecontains the picture compression block 9 for compressing pictures, thesound compression block 10 for compressing sounds, and the code mixingblock 11 for mixing sound codes and picture codes as well as thetransmitter block 12 and the modem 13 which are used to performtransmission of codes. In addition to the aforementioned blocks 1 to 13,the device of FIG. 1 further contains a compression control block 14which controls switching between picture compression and soundcompression.

The sound-picture synchronous compression device of FIG. 1 inputspicture data by means of the picture input block 5 and the camera 6. Inaddition, the device inputs sound data by means of the sound input block7 and the microphone 8. The compression control block 14 switches overexecution of the picture compression block 9 and execution of the soundcompression block 10, so that the picture data and sound data aresubjected to compression respectively. Thus, the picture compressionblock 9 produces compressed picture codes, while the sound compressionblock 10 produces compressed sound codes. The compressed picture codeand compressed sound code are mixed together in a form of a single codeby the code mixing block 11. Then, the mixed codes are subjected totransmission by means of the transmitter block 12 and the modem 13.

FIG. 2 shows a configuration of a sound-picture synchronous reproductiondevice in accordance with the embodiment of the invention, wherein partsequivalent to those of FIG. 13 are designated by the same numerals. Thesound-picture synchronous reproduction device of FIG. 2 is configured bythe device control block 24 for controlling the device as a whole, theCPU 22 for executing programs, the keyboard 23 for inputting commandsand instructions given by the user, and the display 21 for visuallydisplaying states of execution of the programs. In addition, the devicecontains the picture output block 25 and the monitor 26 which are usedto output pictures as well as the sound output block 27 and the speaker28 which are used to output sounds. Further, the device contains thepicture reproduction block 29 for reproducing pictures, the soundreproduction block 30 for reproducing sounds, and the code separationblock 31 for separating the mixed codes into sound codes and picturecodes as well as the receiver block 32 and the modem 33 which are usedto receive the mixed codes transmitted thereto. In addition to theaforementioned blocks 21 to 33, the device of FIG. 2 further contains areproduction control block 34 which controls switching between picturereproduction and sound reproduction.

The sound-picture synchronous reproduction device of FIG. 2 receives themixed codes by means of the receiver block 32 and the modem 33 as soundcodes and picture codes which should be reproduced. So, the codeseparation block 31 separates the mixed codes into the sound codes andpicture codes. The reproduction control block 34 switches over executionof the picture reproduction block 29 and execution of the soundreproduction block 30, so that the picture codes and sound codes aresubjected to reproduction respectively. Thus, picture data correspondingto the reproduced picture codes are output by means of the pictureoutput block 25 and the monitor 26, while the sound data correspondingto the reproduced sound codes are output by means of the sound outputblock 27 and the speaker 28.

Next, a description will be given with respect to operations of thepresent embodiment. FIG. 3 shows a sequence of operations for the soundcompression and picture compression. Steps of FIG. 3 are classified intothree kinds of processing which correspond to the compression control,picture compression and sound compression respectively. So, the stepsare arranged in different columns of FIG. 3 respectively.

In step 81 which belongs to the processing of the compression control,the compression control block 14 designates a number of macro blocks oran amount of data (or codes) to call the picture compression block 9.Herein, the designated number of the macro blocks and the designatedamount of the codes are preset values, which are set in advance in sucha way that the sound and picture synchronize with each other withoutcausing a break-off event in the sound compression. The preset valuesdepend on the system currently used. For example, the preset value canbe determined using the unit of line(s).

After leaving the step 81, the system of the present embodiment proceedsto step 82, wherein the picture compression block 9 performs picturecompression on input picture data. In next step 83, when completing thecompressive encoding with respect to the designated number of macroblocks or the designated amount of codes, the picture compression block9 calls the compression control block 14. The compression control block14 measures a time of the picture compression in step 90. So, in step84, the compression control block 14 calculates a necessary amount ofsound codes based on the measured time of the picture compression so asto call the sound compression block 10. Or, if an amount of sound codeswhich are compressed but are not subjected to transmission is less thana threshold value, the compression control block 14 calculates anecessary amount of sound codes to call the sound compression block 10.

As described above, the step 84 contains two processes which areconnected by “OR”. Next, a description will be given with respect to thebasis of calculations for the necessary amount of sound codes in thelatter process described after “OR” in step 84.

In some case, sound data have been already compressed but remain in atransmission buffer (not shown) of the transmitter block 12 asuntransmitted codes. In such a case, if the system makes a decision thatan amount of the untransmitted codes is less than a prescribed thresholdvalue, it can be assumed that the device has a room for additionaltransmission. Therefore, it is possible to conduct a furtheraccumulation of sound codes in the transmission buffer up to a certainvalue of accumulation which is greater than the threshold value.Therefore, the device performs calculations to produce an amount ofsound codes which can be further accumulated to fill up the transmissionbuffer up to the certain value of accumulation greater than thethreshold value. Therefore, in step 84, the compression control block 14instructs the sound compression block 10 to perform sound compressionwith respect to the calculated amount of sound codes.

After leaving the step 84, the system proceeds to step 85 which belongsto the processing of the sound compression. In step 85, when completingcompressive encoding with respect to the necessary amount of sound codeswhich is designated by the sound compression block 10, the soundcompression block 10 calls the compression control block 14. So, thesystem proceeds to step 86 belonging to the processing of thecompression control. In step 86, the compression control block 14transfers the sound codes and picture codes to the device control block4, which in turn controls the modem 13 to engage transmission of codes.In next step 87, the compression control block 14 designates a number ofmacro blocks or an amount of codes so as to transfer control to thepicture compression block 9. Or, if an amount of sound codes which arecompressed but are not subjected to transmission is greater than athreshold value, the compression control block 14 designates a number ofmacro blocks or an amount of codes so as to transfer control to thepicture compression block 9. So, the system proceeds to step 88belonging to the processing of the picture compression. In step 88, thepicture compression block 9 continues the picture compression. In step89, when completing compressive encoding with respect to the designatednumber of macro blocks or designated amount of codes, the picturecompression block 9 calls the compression control block 14.

According to the operations of the sound-picture synchronous compressiondevice of the present embodiment, every time the encoding is performedwith respect to a certain region of the picture, a decision is made asto whether the sound compression is performed or not, so that only the“necessary” sound compression is performed. Thus, it is possible toperform the sound compression and picture compression at a high speedwithout causing a break-off event of the sound compression.

FIG. 4 shows a sequence of operations of the sound-picture synchronousreproduction device of FIG. 2. Steps of FIG. 4 are classified into threekinds of processing which correspond to the picture reproduction,reproduction control and sound reproduction. So, the steps are arrangedin different columns of FIG. 4 respectively.

At first, the modem 33 receives codes transmitted thereto, so that thereproduction control block 34 receives sound codes and picture codesfrom the device control block 24 in step 91. In step 92, thereproduction control block 34 designates a (prescribed) number of macroblocks or an (prescribed) amount of codes to call the picturereproduction block 29. Thus, the system proceeds to step 93 whichbelongs to the processing of the picture reproduction. In step 93, thepicture reproduction block 29 performs the picture reproduction. In nextstep 94, when completing decoding of the picture codes with respect tothe designated number of macro blocks or the designated amount of codes,the picture reproduction block 29 calls the reproduction control block34. So, the system proceeds to step 95 which belongs to the processingof the reproduction control.

After leaving the step 92, the reproduction control block 34 measures areproduction time of the picture in step 100. Then, the system proceedsto step 95. In step 95, the reproduction control block 34 calculates anecessary amount of sound codes based on the reproduction time of thepicture so as to call the sound reproduction block 30. Or, if an amountof sound codes which are received but are not reproduced is greater thana threshold value, the reproduction control block 34 calculates anecessary amount of sound codes so as to call the sound reproductionblock 30.

As described above, the step 95 contains two processes which areconnected by “OR”. Next, a description will be given with respect to thebasis of calculations for the necessary amount of sound codes in thelatter process described after “OR” in step 95. In some case, a certainamount of sound codes are received but are accumulated in a receivingbuffer (not shown) of the receiver block 32 without being subjected toreproduction. If such an amount of “unreproduced” sound codes is greaterthan a prescribed threshold value, it should be reduced to be less thanthe threshold value. So, the reproduction control block 34 calculates anamount of sound codes being read from the receiving buffer, by which theamount of unreproduced sound codes is reduced to be less than thethreshold value. Therefore, the reproduction control block 34 instructsthe sound reproduction block 30 to perform sound reproduction withrespect to the calculated amount of sound codes in step 95.

After leaving the step 95, the system proceeds to step 96 which belongsto the processing of the sound reproduction. In step 96, the soundreproduction block 30 performs decoding with respect to the designatedamount of sound codes which is designated by the reproduction controlblock 34; then, the sound reproduction block 30 calls the reproductioncontrol block 34. Next, the system proceeds to step 97 which belongs tothe processing of the reproduction control. In step 97, the reproductioncontrol block 34 designates a number of macro blocks or an amount ofcodes so as to transfer control to the picture reproduction block 29.Or, if an amount of sound codes which are received but are notreproduced is less than a threshold value, the reproduction controlblock 34 designates a number of macro blocks or an amount of codes so asto transfer control to the picture reproduction block 29. So, the systemproceeds to step 98 which belongs to the processing of the picturereproduction. In step 98, the picture reproduction block 29 continuesthe picture reproduction. In next step 99, when completing decoding withrespect to the designated number of macro blocks or the designatedamount of codes, the picture reproduction block 29 calls thereproduction control block 34.

According to the aforementioned operations of the sound-picturesynchronous reproduction device of the present embodiment, every timethe decoding is performed with respect to a certain region of thepicture, a decision is made as to whether to perform the soundreproduction or not, so that only the “necessary” sound reproduction isperformed. Thus, it is possible to perform the sound reproduction andpicture reproduction at a high speed without causing a break-off eventin the sound reproduction.

Next, a description will be given with respect to details of theoperations of the sound-picture synchronous compression-reproductionsystem in accordance with the embodiment of the invention. Herein, theoperations of the system are described using flowcharts showing avariety of routines (or programs). Actually, different routines shouldbe provided for compression and reproduction respectively with respectto each of the sound and picture. In the present specification, someroutines are combined together as a single routine to avoid overlappingof the description. As a result, the flowcharts are provided forroutines representing the compression/reproduction control, picturecompression/reproduction and sound compression/reproduction as well ascall subroutines, wherein a slash “/” represents “or” so that thecompression/reproduction control can be interpreted as the compressioncontrol or reproduction control, for example.

For convenience' sake, the description of the flowcharts is made in sucha way that steps of each routine are performed by a unit having acertain name corresponding to its function. For example, the routine ofFIG. 5A is executed by a “compression/reproduction control” unit whichcorresponds to the aforementioned compression control block orreproduction control block.

The routines of FIG. 5A and FIG. 5B are designed to switch over thepicture processing and sound processing by using the call back functionwhich corresponds to an address of a called subroutine. Herein, thecalling side transfers the address to call the subroutine, in which thecall back function is called based on the address.

FIG. 5A shows a flow of steps with respect to the routine of thecompression/reproduction control. In step 101, thecompression/reproduction control unit uses a call for a soundcompression/reproduction unit as a call back function while using anaddress of the call back function and a number of macro blocks or anamount of codes as an argument. Using the call back function as well asthe argument, the compression/reproduction control unit calls a picturecompression/reproduction unit (which corresponds to the aforementionedpicture compression block or picture reproduction block). In step 102, adecision is made as to whether the transmission/receiving operation(i.e., transmission operation or receiving operation) is completed ornot. If the transmission/receiving operation is completed, the systemends execution of the routine of the compression/reproduction controlshown in FIG. 5A. If not, the system proceeds back to the step 101.

FIG. 5B shows a flow of steps with respect to the routine of the picturecompression/reproduction. In step 103, the picturecompression/reproduction unit performs the encoding/decoding operation(i.e., encoding operation or decoding operation) with respect to onemacro block. In step 104, a decision is made as to whether the picturecompression/reproduction unit has completed the encoding/decodingoperation with respect to the designated number of macro blocks or thedesignated amount of codes or not. If the picturecompression/reproduction unit has not completed the aboveencoding/decoding operation, the picture compression/reproduction unitproceeds to step 107. Otherwise, the picture compression/reproductionunit proceeds to step 105 to call the call back function. In step 106,the picture compression/reproduction unit renews the number of macroblocks or the amount of codes. In step 107, a decision is made as towhether the picture compression/reproduction unit has completed thecompression/reproduction operation (i.e., compression operation orreproduction operation) with respect to one frame. If the unit hascompleted the above compression/reproduction operation, the picturecompression/reproduction unit reverts control to the original callingside (i.e., compression/reproduction control unit). If not, the picturecompression/reproduction unit proceeds back to the step 103.

In the case of FIG. 6 and FIG. 7, the system switches over the pictureprocessing and sound processing by stopping the processing until theissuance of an event which corresponds to a synchronization controlsignal of the real-time parallel processing.

FIG. 6 shows a flow of steps with respect to the routine of thecompression/reproduction control. In step 111, thecompression/reproduction control unit uses (or designates) a number ofmacro blocks or an amount of codes as an argument to call the picturecompression/reproduction unit. In step 112, the compression/reproductioncontrol unit waits for occurrence of an event given from the picturecompression/reproduction unit.

In step 113, a decision is made as to whether the event representscompletion of the encoding/decoding operation with respect to thedesignated number of macro blocks or the designated amount of codes. Ifthe event represents the completion of the above encoding/decodingoperation, the compression/reproduction control unit proceeds to step114 to call the sound compression/reproduction unit which corresponds tothe aforementioned sound compression block or sound reproduction block.If not, the compression/reproduction control unit proceeds directly tostep 116. After leaving the step 114, the compression/reproductioncontrol unit proceeds to step 115 so as to issue an event representingcontinuity of the picture compression/reproduction.

In step 116, a decision is made as to whether the event representscompletion of the picture compression/reproduction with respect to oneframe. If the event represents the completion of the above picturecompression/reproduction, the compression/reproduction control unitproceeds to step 117 to make a decision as to whether transmission ofcodes is completed or not. If not, the compression/reproduction controlunit proceeds back to the step 111. If the transmission of the codes isnot completed, the compression/reproduction control unit leaves the step117 to proceed back to the step 111. If the transmission of the codes iscompleted, the compression/reproduction control unit ends execution ofthe routine shown in FIG. 6.

FIG. 7 shows a routine of the picture compression/reproduction. In step118, the picture compression/reproduction unit performsencoding/decoding operation with respect to one macro block. In step119, a decision is made as to whether the encoding/decoding operationhas been completed with respect to the designated number of macro blocksor the designated amount of codes. If the picturecompression/reproduction unit has completed the above encoding/decodingoperation, the picture compression/reproduction unit proceeds to step120 to issue an event representing completion of the encoding/decodingoperation. If not, the picture compression/reproduction unit proceedsdirectly to step 123. After leaving the step 119, the picturecompression/reproduction unit proceeds to step 121 to wait for an eventrepresenting continuity of the picture compression/reproduction. In step122, the picture compression/reproduction unit renews the number ofmacro blocks or the amount of codes.

In step 123, a decision is made as to whether the picturecompression/reproduction has completed with respect to one frame or not.If not, the picture compression/reproduction unit leaves the step 123 toproceed back to the step 118. If the above picturecompression/reproduction has been completed, the picturecompression/reproduction unit proceeds to step 124 to issue an eventrepresenting completion of the picture compression/reproduction. Then,the picture compression/reproduction unit ends execution of the routineof FIG. 7.

FIG. 8, FIG. 9A and FIG. 9B are flowcharts showing subroutines which arecalled during execution of the aforementioned routines.

Specifically, FIG. 8 shows a subroutine for the soundcompression/reproduction. In step 137, the soundcompression/reproduction unit performs the encoding/decoding operationwith respect to one AUU. In step 138, a decision is made as to whetherthe encoding/decoding operation is completed with respect to thedesignated amount of codes or not. If not, the soundcompression/reproduction unit proceeds back to step 137. If the aboveencoding/decoding operation is completed, the soundcompression/reproduction unit transfers control to the calling side.

FIG. 9A shows a subroutine for call 1 of the soundcompression/reproduction, which is an example of a call for the soundcompression/reproduction. In step 131, the system calculates a necessaryamount of sound codes on the basis of a time required for performing thepicture compression/reproduction. Using the calculated amount of thesound codes as an argument, the system calls the sound compression/reproduction unit in step 132. In step 133, if the time required forperforming the picture compress ion/reproduction is longer than athreshold value, the system decreases a number of macro blocks or anamount of codes which is used as an argument. On the other hand, if thetime required for performing the picture compression/reproduction isshorter than the threshold value, the system increases a number of macroblocks or an amount of codes which is used as an argument. Aftercompletion of the step 133, the system reverts control to the callingside.

Incidentally, if the time required for performing the picturecompression/reproduction is 30 ms, the calculated amount of the soundcodes is equivalent to an amount of codes corresponds to a minimumnumber of AAUs which exceed an amount of sound codes corresponding to atime of 30 ms plus α. Herein, α value of a depends on the performance ofthe CPU and/or the communication speed.

In addition, the threshold value as well as an increment or a decrementfor increasing or decreasing the number of macro blocks or the amount ofcodes are adjusted in such a way that the sound compression/reproductionprocessing can be continued without occurrence of a break-off event.They depend on the performance of the CPU and/or the communicationspeed.

FIG. 9B shows a subroutine for call 2 of the soundcompression/reproduction, which is another example of the call for thesound compression/reproduction or call back function. For convenience'sake, FIG. 9B omits a calculation step like the aforementioned step 131which should be provided prior to a decision step 134. In the case ofthe sound compression, a decision is made in step 134, after the abovecalculation step, as to whether an amount of sound codes which arecompressed but are not subjected to transmission is less than athreshold value or not. In the case of the sound reproduction, adecision is made in step 134 as to whether an amount of sound codeswhich are received but are not subjected to reproduction is greater thana threshold value. In step 135, the system uses the amount of codes asan argument to call the sound compression/reproduction unit. Afterleaving the step 135, the system proceeds to step 136. Herein, if a timerequired for performing the picture compression/reproduction is longerthan a threshold value, the system decreases a number of macro blocks oran amount of codes which is used as an argument. On the other hand, ifthe time required for performing the picture compression/reproduction isshorter than a threshold value, the system increases a number of macroblocks or an amount of codes, which is used as an argument. Aftercompletion of the step 136, the system reverts control to the callingside.

Incidentally, if the threshold value is equivalent to a certain amountof sound codes which corresponds to a time of 30 ms, a calculated amountof sound codes is equivalent to a minimum number of AAUs which exceed acertain amount of sound codes, which is calculated as follows:

(Sound codes corresponding to 30 ms)−(remaining sound codes)+α where avalue of α depends on the performance of the CPU and/or thecommunication speed.

Incidentally, the threshold value as well as an increment or a decrementfor increasing or decreasing the number of macro blocks or the amount ofcodes are adjusted in such a way that the sound compression/reproductionprocessing can be continued without occurrence of a break-off event.They depend on the performance of the CPU and/or the communicationspeed.

Lastly, the effects of the invention are summarized as follows:

This invention is designed to reduce the overhead at the switching ofthe parallel operation registers, so it is possible to perform the soundcompression/reproduction and picture compression/reproduction at a highspeed. In addition, the switching is performed at an interval of timewhich is selected in such a way that the sound processing does not breakoff. Further, different modules which separate from each other are usedfor the sound compression/reproduction and picturecompression/reproduction respectively. So, even if a change occurs onthe performance of the CPU and/or the communication speed, it ispossible to adjust the switching timing with ease.

As this invention may be embodied in several forms without departingfrom the spirit of essential characteristics thereof, the presentembodiment is therefore illustrative and not restrictive, since thescope of the invention is defined by the appended claims rather than bythe description preceding them, and all changes that fall within metesand bounds of the claims, or equivalence of such metes and bounds aretherefore intended to be embraced by the claims.

What is claimed is:
 1. A sound-picture synchronous compression devicecomprising: sound compression means for compressing sound information;picture compression means for compressing picture information; andcontrol means for controlling the sound compression means and thepicture compression means in such a way that the sound information andthe picture information are respectively compressed in a synchronizedmanner, wherein the control means further comprises: picture compressioninstruction means for designating a prescribed amount of pictureinformation in a picture compression instruction that is supplied to thepicture compression means; time measurement means for measuring a timethat the picture compression means takes to perform picture compressionon the prescribed amount of picture information in response to thepicture compression instruction and without reference to said soundinformation; calculation means for calculating an amount of soundinformation which can be compressed by the sound compression meanswithin the measured time; and sound compression instruction means fordesignating the calculated amount of sound information in a soundcompression instruction that is supplied to the sound compression means.2. The device according to claim 1, wherein the control means furthercomprises: means for instructing the picture compression means tocontinue the picture compression.
 3. A sound-picture synchronousreproduction device comprising: sound reproduction means for reproducingcompressed sound information; picture reproduction means for reproducingcompressed picture information; and control means for controlling thesound reproduction means and the picture reproduction means in such away that the compressed sound information and the compressed pictureinformation are respectively reproduced in a synchronized manner[.],wherein the control means further comprises: picture reproductioninstruction means for designating a prescribed amount of compressedpicture information in a picture reproduction instruction that issupplied to the picture reproduction means; time measurement means formeasuring a time that the picture reproduction means takes to performpicture reproduction process on the prescribed amount of compressedpicture information in response to the picture reproduction instructionand without reference to said compressed sound information; calculationmeans for calculating an amount of compressed sound information whichcan be reproduced by the sound reproduction means within the measuredtime; and sound reproduction instruction means for designating thecalculated amount of compressed sound information in a soundreproduction instruction that is supplied to the sound reproductionmeans.
 4. The device according to claim 3, wherein the control meansfurther comprises: means for instructing the picture reproduction meansto continue the picture reproduction.
 5. A sound-picture synchronouscompression method which compresses sound information and pictureinformation respectively in a synchronized manner, the method comprisingthe steps of: designating a prescribed amount of picture information toperform picture compression; measuring a time taken to compress theprescribed amount of picture information without reference to soundinformation; calculating an amount of sound information which can becompressed within the measured time; and performing sound compression onthe calculated amount of sound information.
 6. The method according toclaim 5, further comprising the step of: continuing the picturecompression after completion of the sound compression.
 7. Asound-picture synchronous reproduction method which reproducescompressed sound information and compressed picture information in asynchronized manner, the method comprising the steps of: designating aprescribed amount of compressed picture information to perform picturereproduction; measuring a time taken to reproduce the prescribed amountof compressed picture information without reference to soundinformation; calculating an amount of compressed sound information whichcan be reproduced within the measured time; and performing soundreproduction on the calculated amount of compressed sound information.8. The method according to claim 7, further comprising the step of:continuing the picture reproduction after completion of the soundreproduction.
 9. A sound-picture synchronous compression devicecomprising: picture input means for inputting picture data to producepicture codes; sound input means for inputting sound data to producesound codes; designation means for designating an amount of picturecodes which correspond to a numbers of macro blocks; picture compressionmeans for performing picture compression on the designated amount ofpicture codes; calculation means for calculating an amount of soundcodes which can be compressed within a time taken to compress thedesignated amount of picture codes, said time determined withoutreference to sound codes; sound compression means for performing soundcompression on the calculated amount of sound codes; code mixing meansfor mixing compressed picture codes and compressed sound codes together;and transmission means for transmitting mixed codes.
 10. A sound-picturesynchronous reproduction device comprising: receiving means forreceiving mixed codes transmitted thereto; code separation means forseparating the mixed codes into compressed picture codes and compressedsound codes; designation means for designating an amount of thecompressed picture codes which correspond to a number of macro blocks;picture reproduction means for performing picture reproduction on thedesignated amount of the compressed picture codes; calculation means forcalculating an amount of the compressed sound codes which can bereproduced within a time taken to reproduce the designated amount of thecompressed picture codes, said time determined without reference tosound codes; sound reproduction means for performing sound reproductionon the calculated amount of the compressed sound codes; and output meansfor outputting pictures and sounds on the basis of reproduced picturecodes and reproduced sound codes.